[AQUA] CLI to verify policies to perform operations related to AQUA. #1218

amitkrprajapati · 2025-06-30T15:01:35Z

This utility is designed to help users verify whether they have the necessary permissions to perform common AQUA workflows, such as model registration, deployment, evaluation, and fine-tuning.

Common Methods:
ads aqua verify_policies common_policies
ads aqua verify_policies model_register
ads aqua verify_policies model_deployment
ads aqua verify_policies evaluation
ads aqua verify_policies finetune

Help:
ads aqua verify_policies -- --help

github-actions · 2025-06-30T15:01:44Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2025-06-30T15:32:04Z

📌 Cov diff with main:

📌 Overall coverage:

mrDzurb · 2025-07-03T00:18:35Z

Could you add description to this PR?

github-actions · 2025-07-03T17:21:00Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2025-07-03T17:50:10Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2025-07-04T08:48:53Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2025-07-04T09:18:00Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2025-07-08T22:37:30Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

mrDzurb · 2025-07-08T22:38:43Z

ads/aqua/verify_policies/entities.py

+    UNVERIFIED = "UNVERIFIED"
+
+
+@dataclass(repr=False)


NIT: We are trying to move everything to pydantic classes instead.

These classes are inheriting from DataClassSerializable. Do we have something similar with Pydantic.

Yeap we do have

https://github.com/oracle/accelerated-data-science/blob/main/ads/aqua/config/utils/serializer.py

mrDzurb · 2025-07-08T22:40:00Z

ads/aqua/verify_policies/messages.py

+    utils.list_compartments.__name__: {
+        "name": "List Compartments",
+        "error": "Unable to retrieve the list of compartments. Please verify that you have the required permissions to list compartments in your tenancy. ",
+        "policy_hint": "Allow dynamic-group aqua-dynamic-group to inspect compartments in tenancy"


As we discussed during the syncup, let's also try to add a link to the GH docs, where the policies can be setup.

Added a log: logger.info(f"Refer to: {POLICY_HELP_LINK}")

This log will be printed everytime when a operation fails.
POLICY_HELP_LINK: https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/ai-quick-actions/policies/README.md

mrDzurb · 2025-07-08T22:41:58Z

ads/aqua/verify_policies/utils.py

+        subnet_id = kwargs.pop("subnet_id", None)
+        job_infrastructure_type = "STANDALONE" if subnet_id is not None else "ME_STANDALONE"
+
+        response = self.aqua_model.ds_client.create_job(


NIT: This could be done using ADS Jobs implementation.
https://accelerated-data-science.readthedocs.io/en/latest/user_guide/jobs/run_python.html

In current implementation, I am creating Job with shell environment and running a simple "echo" command. I am directly uploading the artifact data without saving the file locally.
In job run: I am waiting for run to complete, by constantly monitoring lifecycle state.

I found ads.job little untidy to achieve the same.

mrDzurb · 2025-07-08T22:45:05Z

Will the ads aqua verify_policies --help show the guidance, how the verifier can be used, if not, then we should add this.
@amitkrprajapati

github-actions · 2025-07-08T23:05:39Z

📌 Cov diff with main:

📌 Overall coverage:

VipulMascarenhas

added some minor comments

VipulMascarenhas · 2025-07-09T02:59:53Z

ads/aqua/verify_policies/utils.py

+    using Oracle Accelerated Data Science (ADS) SDK.
+    """
+    def __init__(self):
+        self.aqua_model = AquaModelApp()


nit: to keep it better readable/maintainable, might be good to load most the below clients directly from AquaApp class, and use aqua_model for testing individual functions. ideally, clients should have been reused in the child classes in aqua, but we've done it on few instances and not all.

Some of the list operations use limit=3, so they fetch data directly from the OCI client. If I change the implementation to ads specific, it tries to list all resources even if it has multiple pages. For policy verification, a list operation with a limit is enough.

VipulMascarenhas · 2025-07-09T03:05:13Z

ads/aqua/verify_policies/utils.py

+        model_id = self.aqua_model.ds_client.create_model(create_model_details=create_model_details).data.id
+        self.aqua_model.ds_client.create_model_artifact(
+            model_id=model_id,
+            model_artifact=b"7IV6cktUGcHIhur4bXTv"


nit: use a constant for this, also add comment on what this string means.

Moved to CONSTANT variable. This is dummy model_artifact bytes.

VipulMascarenhas · 2025-07-09T03:05:32Z

ads/aqua/verify_policies/utils.py

+        self.job_id = None
+        self.limit = 3
+
+    def list_compartments(self, **kwargs):


add docstrings for all functions.

VipulMascarenhas · 2025-07-09T03:09:09Z

log.txt

@@ -0,0 +1,24 @@
+👋 Waiting, Amit!


does this file need to be included?

…celerated-data-science into ODSC-72417/Policy_Verification

github-actions · 2025-07-09T10:18:43Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2025-07-09T10:50:28Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2025-07-10T16:28:49Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2025-07-10T16:58:12Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2025-07-10T19:57:42Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2025-07-10T20:26:40Z

📌 Cov diff with main:

📌 Overall coverage:

VipulMascarenhas

lgtm 👍

CLI to verify policies to perform operations related to AQUA.

8c50bcb

amitkrprajapati requested review from darenr, mayoor, mrDzurb, VipulMascarenhas, qiuosier and ahosler as code owners June 30, 2025 15:01

oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Jun 30, 2025

mrDzurb changed the title ~~CLI to verify policies to perform operations related to AQUA.~~ [AQUA] CLI to verify policies to perform operations related to AQUA. Jun 30, 2025

amitkrprajapati added 2 commits July 3, 2025 22:22

Unint Test and documentation

d3932b9

fix for create mvs

201a3c4

Add subnet

979e52b

amitkrprajapati and others added 2 commits July 4, 2025 19:25

Fix Job run for finetune.

5a27cda

Merge branch 'main' into ODSC-72417/Policy_Verification

bb1330c

mrDzurb reviewed Jul 8, 2025

View reviewed changes

VipulMascarenhas reviewed Jul 9, 2025

View reviewed changes

amitkrprajapati added 2 commits July 9, 2025 15:46

Address PR Comments

cfc5c45

Merge branch 'ODSC-72417/Policy_Verification' of github.com:oracle/ac…

91e84f5

…celerated-data-science into ODSC-72417/Policy_Verification

Merge branch 'main' into ODSC-72417/Policy_Verification

b04da19

amitkrprajapati requested review from mrDzurb and VipulMascarenhas July 10, 2025 17:21

mrDzurb approved these changes Jul 10, 2025

View reviewed changes

Merge branch 'main' into ODSC-72417/Policy_Verification

a34675b

VipulMascarenhas approved these changes Jul 10, 2025

View reviewed changes

mrDzurb merged commit 343f7e1 into main Jul 10, 2025
25 checks passed

[AQUA] CLI to verify policies to perform operations related to AQUA. #1218

[AQUA] CLI to verify policies to perform operations related to AQUA. #1218

Uh oh!

Conversation

amitkrprajapati commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 30, 2025

Uh oh!

github-actions bot commented Jun 30, 2025

Uh oh!

mrDzurb commented Jul 3, 2025

Uh oh!

github-actions bot commented Jul 3, 2025

Uh oh!

github-actions bot commented Jul 3, 2025

Uh oh!

github-actions bot commented Jul 4, 2025

Uh oh!

github-actions bot commented Jul 4, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrDzurb commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

VipulMascarenhas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

github-actions bot commented Jul 10, 2025

Uh oh!

github-actions bot commented Jul 10, 2025

Uh oh!

github-actions bot commented Jul 10, 2025

Uh oh!

github-actions bot commented Jul 10, 2025

Uh oh!

VipulMascarenhas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

amitkrprajapati commented Jun 30, 2025 •

edited

Loading

mrDzurb commented Jul 8, 2025 •

edited

Loading